Papers with NLP community values
Utility is in the Eye of the User: A Critique of NLP Leaderboards (2020.emnlp-main)
Copied to clipboard
| Challenge: | Performance-based evaluation has been at the expense of other attributes valued by the NLP community, such as compactness and energy efficiency. |
| Approach: | They propose to frame both the leaderboard and NLP practitioners as consumers and the benefit they get from a model as its utility to them. |
| Outcome: | The proposed model size and energy efficiency benchmarks have been successful in driving the creation of more accurate models, but have been at the expense of other attributes valued by the NLP community. |